feat(c/driver_manager): add connection profile interface #3876

zeroshade · 2026-01-09T20:03:36Z

An attempt to implement the ideas as suggested in #3765 to provide the ability to specify and load connection profiles to define options for use by the driver manager.

This creates an AdbcConnectionProfile struct and defines an AdbcConnectionProfileProvider function pointer typedef to allow for customized management of profiles. This also implements a default file-based profile provider as described in #3765 (comment) which will be used if no custom provider has been set.

This allows easy expansion in the future for non-file-based connection profile providers while still implementing the easier case of using file-based profiles, including hierarchical specification for now. See the documentation comments added to adbc_driver_manager.h for the full description of the semantics and explanation.

zeroshade · 2026-01-09T20:04:58Z

CC @davidhcoe

if there's anyone I missed please feel free to tag others that might be interested in looking this over and reviewing it. Once everyone is onboard with the design and ideas I'll implement the handling of these profiles for the other language bindings.

paleolimbot

Cool!

All just questions from me to take or leave 🙂

paleolimbot · 2026-01-09T22:06:21Z

c/include/arrow-adbc/adbc_driver_manager.h

+/// version = 1
+/// driver = "driver_name"
+///
+/// [options]
+/// option1 = "value1"
+/// option2 = 42
+/// option3 = 3.14


Do you want to wrap this in a top level [profile] item (like pyproject.toml's [project] or Cargo.toml's [workspace])?

Suggested change

/// version = 1

/// driver = "driver_name"

///

/// [options]

/// option1 = "value1"

/// option2 = 42

/// option3 = 3.14

/// [adbc.profile]

/// version = 1

/// driver = "driver_name"

///

/// [options]

/// option1 = "value1"

/// option2 = 42

/// option3 = 3.14

Personally I was trying to avoid having to do that. Is there any benefit to doing so?

Just seemed like a common idiom in .toml configs, perhaps to allow for future expansion (e.g., if you ever want >1 profile to live in one .toml)

i think after a lot of discussion between myself, @lidavidm and @ianmcook we came to the decision that we don't want more than 1 profile to live in one .toml file haha 😄

Having multiple connection profiles defined in a single TOML file could potentially be nice, but it complicates the profile search procedure, and I worry that it could confuse users.

c/include/arrow-adbc/adbc_driver_manager.h

paleolimbot · 2026-01-09T22:19:53Z

c/include/arrow-adbc/adbc_driver_manager.h

+  AdbcStatusCode (*GetDriverName)(struct AdbcConnectionProfile* profile,
+                                  const char** driver_name, struct AdbcError* error);


Suggested change

AdbcStatusCode (*GetDriverName)(struct AdbcConnectionProfile* profile,

const char** driver_name, struct AdbcError* error);

AdbcStatusCode (*GetDriverName)(struct AdbcConnectionProfile* profile,

const char** driver_name, AdbcDriverInitFunc* init_func, struct AdbcError* error);

Is there any value to allowing a profile to optionally specify the init function directly? (R could maybe use this to find R package versions of drivers)

The way the code is set up doesn't allow for this. If there is an init func set then the profile handling is currently entirely skipped and it's up to the init_func to handle everything. Given the way we envision profiles working, I don't think that it makes sense for the profile to set or specify the init function.

I think you'd just set args->init_func instead of args->driver (understood if you'd prefer not to)

I personally prefer not, but I'm open to it if others think it's worthwhile.

go/adbc/drivermgr/adbc_driver_manager.cc

davidhcoe · 2026-01-09T23:26:37Z

@CurtHagenlocher will be interested in this as well.

lidavidm · 2026-01-12T02:24:45Z

docs/source/format/connection_profiles.rst

+Overview
+========
+
+Similar to ODBC's ``odbc.ini``, the ADBC driver manager supports **connection profiles**


nit, but I think we should explain this as its own thing, and cite ODBC as an inspiration further down

Agreed. How about we structure the beginning of this page similar to how the ADBC Driver Manager and Manifests docs page is structured? Like:

There are two ways to pass connection options to driver managers:

Directly specifying connection options as arguments to driver manager functions in your application code.

Referring to a connection profile which contains connection options.

(or something like that)

And we could add a section that briefly explains method 1, before continuing to the content here which covers method 2.

lidavidm · 2026-01-12T02:26:32Z

docs/source/format/connection_profiles.rst

+   driver = "adbc_driver_snowflake"
+
+   [options]
+   adbc.snowflake.sql.account = "env_var(SNOWFLAKE_ACCOUNT)"


Hmm. Why not use option = { env_var = "NAME" }?

That would prevent us from using environment variables to define substrings, which we will need for the drivers that expect their connection arguments to be in a URI.

uri = "scheme://host:port?user=user&pw=env_var(PASSWORD)

Should we consider using ${} instead of env_var()?

That is more concise and makes escaping literals easier (e.g. with \$ or $$).

I think ${} might be a little more familiar, unless you expect to eventually have other things besides env_var.

I believe the expectation was that we would eventually have more special functions in addition to env_var which is why this was the original suggestion. For eventual consistency

Upon further consideration, I think env_var is good, because it's clear and explicit, and because (as Matt says) it will be more consistent with future functions that we might want to add, for example to retrieve credentials from a secrets management tool.

lidavidm · 2026-01-12T02:29:17Z

c/include/arrow-adbc/adbc_driver_manager.h

+  /// \brief Release the profile and perform any cleanup.
+  void (*release)(struct AdbcConnectionProfile* profile);


(Does the manager call release or does the application?)

The manager calls release currently, in general the application will never interact with profile objects unless they explicitly want to.

zeroshade · 2026-01-14T19:00:44Z

@CurtHagenlocher @davidhcoe @lidavidm @ianmcook any further comments?

ianmcook · 2026-01-15T05:20:46Z

docs/source/format/connection_profiles.rst

+
+   AdbcDatabase database;
+   AdbcDatabaseNew(&database, &error);
+   AdbcDatabaseSetOption(&database, "profile", "my_snowflake_prod", &error);


@lidavidm Similar to what I added in af734d3, should we also add a warning like this somewhere in the docs?

Driver managers may treat some option keys as manager-reserved and handle them without forwarding them to the underlying driver. In particular, the option key profile is reserved for connection profiles and must not be implemented or interpreted by drivers.

Yes, I think that's reasonable

Where's the best place to add this, @lidavidm ?

Maybe the docstring for AdbcDatabaseSetOption in adbc.h is best.

Added in 7b8d420.

@lidavidm Does this comment qualify as a sufficiently small change to adbc.h that this would not be considered a change to the ADBC API spec? I hope so.

zeroshade added 3 commits January 9, 2026 14:57

feat(c/driver_manager): add connection profile interface

ca407d0

pre-commit linting

d52726a

typos

cb5af50

zeroshade requested a review from lidavidm as a code owner January 9, 2026 20:03

zeroshade requested review from CurtHagenlocher, felipecrv, ianmcook, kou and paleolimbot January 9, 2026 20:03

github-actions bot modified the milestone: ADBC Libraries 23 Jan 9, 2026

zeroshade added 9 commits January 9, 2026 15:06

darn you windows

11f7c73

update test cases for ParseDriverURI

41970e8

darn you windows

8c6166c

fix heap issue

43ac82b

some test fixes

543b264

more test fix

e914f38

one more test leak

e81407a

windows sucks

b384b4c

pre-commit

d145620

paleolimbot reviewed Jan 9, 2026

View reviewed changes

zeroshade added 3 commits January 9, 2026 17:49

fixes from comments and nested tables

dd3deaf

add test using env_var()

9e75ffd

pre-commit

de7e5fd

paleolimbot reviewed Jan 9, 2026

View reviewed changes

go/adbc/drivermgr/adbc_driver_manager.cc Outdated Show resolved Hide resolved

maybe clang will like this better

29e87bd

zeroshade added 2 commits January 9, 2026 18:39

maybe clang will like me now

1bb0569

extra complexity to make clang happy?

87674d6

zeroshade added 4 commits January 9, 2026 18:52

add docs

391f648

yet another attempt

2caa736

fix remaining issues, i hope

2a2ff0f

this might fix the issue?

835506c

lidavidm reviewed Jan 12, 2026

View reviewed changes

Disallow profile as a driver manifest basename

af734d3

ianmcook reviewed Jan 15, 2026

View reviewed changes

Reserve the profile option key for use by driver managers

7b8d420

		AdbcStatusCode (GetDriverName)(struct AdbcConnectionProfile profile,
		const char** driver_name, struct AdbcError* error);

		/// \brief Release the profile and perform any cleanup.
		void (release)(struct AdbcConnectionProfile profile);

feat(c/driver_manager): add connection profile interface #3876

Are you sure you want to change the base?

feat(c/driver_manager): add connection profile interface #3876

Uh oh!

Conversation

zeroshade commented Jan 9, 2026

Uh oh!

zeroshade commented Jan 9, 2026

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

davidhcoe commented Jan 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zeroshade Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zeroshade commented Jan 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zeroshade Jan 15, 2026 •

edited

Loading